Papers with interpretability methods
Influence Functions for Sequence Tagging Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Named Entity Recognition, Part-of-Speech tagging, and Semantic Role Labeling are standard tasks in NLP, but there has been little work on interpretability methods for sequence taging. |
| Approach: | They propose to extend influence functions to sequence tagging tasks by identifying noisy annotations in NER corpora. |
| Outcome: | The proposed methods are able to identify noisy annotations in NER corpora and are scalable. |
Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) are a promising tool for disentangling FM representations, but they struggle to capture rare, yet crucial concepts in the data. |
| Approach: | They propose a technique to train Sparse Autoencoders to illuminate elusive dark matter features by focusing on specific subdomains. |
| Outcome: | The proposed method achieves 12.5% better classification accuracy than general-purpose SAEs when applied to remove spurious gender information. |
Attention Consistency for LLMs Explanation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing interpretability methods face limitations such as low resolution and high computational cost. |
| Approach: | They propose a multi-layer attention consistency score to estimate the importance of input tokens in large language models. |
| Outcome: | The proposed heuristic achieves a favorable trade-off between interpretability quality and computational efficiency . |
Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing work identifies task-specific shortcuts via human priors or error analyses, which requires extensive expertise and efforts. |
| Approach: | They propose to automatically identify spurious correlations in NLP models at scale by using existing interpretability methods to extract tokens that significantly affect model’s decision process. |
| Outcome: | The proposed method can identify spurious correlations in NLP models at scale and mitigate these leads to more robust models in multiple applications. |
A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent instruction fine-tuned models can solve multiple NLP tasks when prompted to do so, but current research focuses on standard performance benchmarks, leaving compelling fairness and ethical considerations behind. |
| Approach: | They propose a method to mitigate gender bias in machine translation by using a corpus of machine translations from the WinoMT corpus. |
| Outcome: | The proposed model can solve multiple NLP tasks when prompted, but it lacks fairness and ethical considerations. |
DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformers (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing interpretability methods have been proposed to interpret the inner workings of Transformer models at different levels of precision and complexity. |
| Approach: | They propose a method to analyze encoder-decoder Transformers by using the decoder module Model Output encoder to cross-attend representations of intermediate encoder activations instead of using the default output. |
| Outcome: | The proposed method maps uninterpretable representations to human-interpreted sequences of words or symbols, shedding new light on the information flow in this popular but understudied class of models. |
Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals (2024.acl-long)
Copied to clipboard
| Challenge: | Existing interpretability research focused on analyzing a single mechanism . et al., 2023) focused on finding how models copy or recall factual knowledge . |
| Approach: | They propose a competition of mechanisms that focuses on the interplay of multiple mechanisms instead of individual mechanisms . they uncover how and where the competition of mechanism happens within LLMs using logit inspection and attention modification methods. |
| Outcome: | The proposed model is based on two interpretability methods, logit inspection and attention modification. |
RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods to disentangle individual neurons from multiple high-level concepts are not yet benchmarked. |
| Approach: | They propose a method of Multi-task Distributed Alignment Search that allows to find distributed representations satisfying multiple causal criteria. |
| Outcome: | The proposed method achieves state-of-the-art on the target language model with Llama2-7B . |
Forecasting Earnings Surprises from Conference Call Transcripts (2023.findings-acl)
Copied to clipboard
| Challenge: | Earnings conference calls contain over 5,000 words of text and large amounts of industry jargon . this length and domain-specific language present problems for generic pretrained language models. |
| Approach: | They propose a task of predicting earnings surprises from earnings call transcripts and propose linguistic models that use a long document dataset to test financial understanding. |
| Outcome: | The proposed model can predict earnings surprises from earnings conference calls with reasonable accuracy and shows that it is possible to interpret the data with different interpretability methods. |
Easy to Decide, Hard to Agree: Reducing Disagreements Between Saliency Methods (2023.findings-acl)
Copied to clipboard
| Challenge: | A popular approach to unveiling the black box of neural NLP models is to leverage saliency methods, which assign scalar importance scores to each input component. |
| Approach: | They propose to use saliency methods to evaluate whether an explanation is faithful and argue that Pearson-r is a better-suited alternative to rank correlation. |
| Outcome: | The proposed methods exhibit weak rank correlations even when applied to the same model instance and advocated for alternative diagnostic methods. |
CREST: A Joint Framework for Rationalization and Counterfactual Text Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for analyzing and training NLP models have not been integrated to combine their complementary advantages. |
| Approach: | They introduce a framework for selective rationalization and counterfactual text generation that leverages CREST to regularize selective rationales and a loss function that regularizes selective rationals. |
| Outcome: | The proposed framework generates valid counterfactuals that are more natural than those produced by previous methods and can be used for data augmentation at scale. |
Model Interpretability and Rationale Extraction by Input Mask Optimization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for creating explanations for black-box models struggle with deriving easily interpretable explanations. |
| Approach: | They propose a model-agnostic method to generate extractive explanations for neural network predictions using masking parts of the input that the model does not consider indicative of the respective class. |
| Outcome: | The proposed method achieves state-of-the-art results in a paragraph-level rationale extraction task, showing that this task can be performed without training a specialized model. |
Analyzing Transformers in Embedding Space (2023.acl-long)
Copied to clipboard
| Challenge: | Understanding Transformer-based models has attracted significant attention . a zero-pass approach is feasible for some parameters, and for two-layer attention networks . |
| Approach: | They propose a theoretical framework where parameters of a trained Transformer are interpreted by projecting them into the embedding space. |
| Outcome: | The proposed framework shows that pre-trained and fine-tuned models can be interpreted in embedding space. |
Bridging Internal Consistency and External Alignment: A Causal and Dynamic Interpretability Framework for LLM Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing interpretability methods focus on internal and external aspects of the model . existing explanations often focus on surface correlations or static dependencies . |
| Approach: | They propose a causal and dynamic interpretability framework for Large Language Models . they characterize backdoor-adjusted causal effects of generated prefix and prompt . |
| Outcome: | The proposed framework provides a unified causal view of internal consistency and external alignment in LLM generation dynamics. |
Fact Recall, Heuristics or Pure Guesswork? Precise Interpretations of Language Models for Fact Completion (2025.findings-acl)
Copied to clipboard
| Challenge: | Language models (LMs) can make a correct prediction based on many possible signals in a prompt, but not all corresponding to recall of factual associations. |
| Approach: | They propose a model-specific recipe for constructing datasets with examples of four different prediction scenarios: generic language modeling, guesswork, heuristics recall and exact fact recall. |
| Outcome: | The proposed model-specific recipe yields distinct results for each scenario. |
DEFT: Demystifying VLN Failures via a Unified Dual-View Explainability Framework for LLM-based Agents (2026.acl-long)
Copied to clipboard
| Challenge: | Existing interpretability methods isolate temporal criticality from feature salience, creating an alignment gap and failing to account for the behavioral instability of black-box agents. |
| Approach: | They propose a unified dual-view framework that jointly analyzes when a decision is pivotal and what visual evidence grounds it. |
| Outcome: | Extensive experiments on MatterPort3D show that DEFT outperforms baselines in both temporal and feature fidelity. |
TinySQL: A Progressive Text-to-SQL Dataset for Mechanistic Interpretability Research (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing text-to-SQL datasets are too complex and noisy for rigorous interpretability analysis. |
| Approach: | They propose text-to-SQL generation as an ideal task to study mechanistic interpretability . they use edge attribution patching and sparse autoencoders to identify minimal circuits . |
| Outcome: | The proposed task combines the formal structure of toy tasks with real-world complexity. |